Dalí: A High Performance Main Memory Storage Manager

Performance needs of many database applications dictate that the entire database be stored in main memory. The Dali system is a main memory storage manager designed to provide the persistence, availability and safety guarantees one typically expects from a diskresident database, while at the same time providing very high performance by virtue of being tuned to support in-memory data. Dali follows the philosophy of treating all data, including system data, uniformly as database files that can be memory mapped and directly accessed/updated by user processes. Direct access provides high performance; slower, but more secure, access is also provided through the use of a server process. Various features of Dali can be tailored to the needs of an application to achieve high performance for example, concurrency control and logging can be turned off if not desired, which enables Dali to efficiently support applications that require non-persistent memory resident data to be shared by multiple processes. Both objectoriented and relational databases can be implemented on top of Dali. Permisrion io copy wiihoui fee all OT pari of this material ir granied provided thai the copies are not made 01 diriribuied for direci cornmenial advantage, the VLDB copyright notice and the title of the publication and ii8 dale appeur, and notice ia given thaf copying ir by pennirrion of the Very Large Data Bare Endowment. To copy otherwire, or to republish, requires a fee and/or special pennirrion from the Endowment. Proceedings of the 20th VLDB Conference Santiago, Chile, 1994 There are a number of database applications, particularly in the telecommunications industry, where very high performance access to data is required. Such applications typically require high transaction rates, coupled with very low latency for transactions, and stringent durability and availability requirements. As as example, consider a real phon+company application where phone call data is recorded, and queries against the data can be issued. The application requires several thousand (albeit small) requests (lookups/updates) to be processed per second, with less than 50 milliseconds latency for lookups, and less than a few minutes of downtime a year. Such applications have been previously implemented as stand alone programs that run in main memory and provide their own (usually limited) forms of sharing and persistence mechanisms. It is increasingly being realized that the core requirements of these type of applications would best be met by using an underlying main-memory storage manager that supports full functionality such it9 transaction management, concurrency control and recovery services. Using the same storage manager across multiple applications can greatly reduce development costs. The increasing availability of large and relatively cheap memory also suggests that more database applications could reside entirely or almost entirely in main memory. Such applications will experience performance benefits by having data cached in main memory. However, if the storage manager supporting such applications is tailored to main memory, significant additional performance benefits can be achieved, as shown in [LSC92]. Thus, storage managers tailored to main memory would also be ideally suited for such databases. The Dali’ system, implemented at AT&T Bell Laboratories, is a storage manager for persistent data that has been optimized for environments in which the database is main-memory resident. Dali uses a memory-mapped architecture, where the database is mapped into the virtual address space of the process. While Dali can be used in systems where the database is larger than main-memory, the architecture of Dali, in particular its recovery mechanism, has been designed to deliver high performance when the database fits into main memory. Parts of the Dali architecture that differ from storage manager architectures for disk-based systems include the following: l The use of the recovery mechanism of [JSS93], which is tailored to ‘high-performance mainmemory databases. l The partition of the database into database files that can be memory mapped into shared memory by user processes. The benefits of this are: Data can be accessed directly by user processes, avoiding costly interprocess communication. User processes can memory map selected database files rather than the whole database, enabling users to map into memory only the parts of the database relevant to the application. System data, such as log records and lock structures, are themselves stored in a system database file, and can be memory mapped and manipulated directly by user processes. Multiple processes can concurrently access the database. l Tolerance to software faults. Fault-tolerance features are particularly important in the context of Dali since processes may access and update shared data directly for performance reasons. The faulttolerance features provided by Dali include recovery from processes that fail while updating shared data, and detection and recovery from corruption of shared data. l Tunability of the system, which enables tailoring of the database to achieve high performance. Dali allows various features, such as concurrency control and logging, to be turned off if not required. Users can also control the partition of Dali system code between libraries linked with the user process, with direct access to the database, and code executed at server processes. 1 Named in honor of Salvadore Dali, for his famous painting, “The Per&tencc of Memory”. 0 Support for compression of individual data ‘items’. Data items are structured as a fixed size header and a variable sieed data component, which facilitates data compression. Dali provides many other useful features. For instance, high availability is achieved through support for remote backups and hot spares and support for online schema and software changes. By providing direct access to data, Dali is weil suited for a “client as server” environment [CD+94], where users add software on top of the storage manager to build a server with a higher level of functionality. Dali also provides an interface to access data from remote sites, based on remote procedure calls. Coupled with the ability to split the database into files, this feature can be used to implement a network of Dali systems in case the sise of the database is larger than the maximum physical memory availability on a machine. The architecture of the Dali storage manager is not tailored specifically to a particular data model. Thus, it is possible to build relational as well as objectoriented database systems on top of Dali. A prototype of Dali is operational, and incorporates most of the features described in this paper. A prototype of MM-Ode [GLS94], a main-memory version of the Ode object-oriented database system [AG89], has been built on top of the Dali. We are in the process of implementing a relational interface as well, on top of Dali.